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(54) An apparatus for manipulating an object displayed on a display device 

played object. 



(57) An apparatus for use with a display device to 
manipulate an object displayed on the display device. 
The apparatus comprises a touch panel (1 1), mounted 
to the display device (3) and representing a display sur- 
face of the display device, which is sensitive to charac- 
teristics of a touching contact on the touch panel, the 
characteristics including at least co-ordinate positions of 
the touching contact, for outputting corresponding touch 
panel information representing the characteristics of the 
touching contact on the touch panel; and object manip- 
ulation means (50-52) for manipulating and displaying 
the object of the display device in response to the touch 
panel information. The object manipulation means com- 
prises object storage means (7) for storing at least one 
data file which stores object data for displaying the 
object; display information storage means (1T) for stor- 
ing display position information which specifies the posi- 
tion where the object is displayed on the display device 
(3); and display control means (52) for recognizing a 
"pick" manipulation of the object when the touch panel 
information indicates a movement such that two touch- 
ing contacts touch both sides of the object, move and 
stop with a distance in between, and for controlling to 
display the object on the display device accordingly. 
This enables a user easily to pick up and move a dis- 
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Description 

[0001] The present invention relates to an apparatus 
for manipulating an object displayed on a display 
device. 5 
[0002] As use of computer systems for data process- 
ing has become widespread in recent years, more and 
more users are being required to input data to and con- 
verse with data processors such as work stations and 
personal computers. A vast range of application pro- 10 
grams are available for recent data processors and even 
a complicated application can be processed by using 
such application programs in combination. However, 
there is a problem that such data processors are very 
difficult to handle, especially to manipulate an object 15 
displayed on a display device, for those who have little 
knowledge of computers. 

[0003] Therefore, an apparatus for manipulating an 
object displayed on a display device, which is easy to 
use even for a person who has no special knowledge of 20 
computers, is in great demand. 
[0004] Fig. 1 illustrates a computer system with a con- 
ventional user interface. 

[0005] A computer system with a conventional user 
interface consists mainly of a central processing unit 25 
(CPU) 4, a main memory 5, a keyboard/mouse 2, a 
frame memory 60 and a hard disk interface 71 , which 
are interconnected via a system bus interface, and also 
a hard disk 7 and a display unit 3, which are connected 
to system bus interface via the hard disk interface and 30 
the frame memory 6, respectively. The main memory 5 
stores a system control program and application pro- 
grams which handle graphics processing, and provides 
a work area for use by the programs. The CPU 4 per- 
forms display operations under control of the programs. 35 
The hard disk 7 stores a data file for graphics to be dis- 
played on the display unit 3. The frame memory 6 stores 
a frame of picture (or object) data to be displayed on the 
display unit 3. 

[0006] To manipulate an object displayed on a display 40 
unit 3 in the above system, an operator is required to 
input a command for manipulating the object by using a 
keyboard/mouse 2 or to select an icon (a symbolic rep- 
resentation of a computer function) displayed on a dis- 
play unit 3 by using the keyboard/mouse 2 in order to 45 
command a desired function. However, it is troublesome 
and annoying to use a keyboard/mouse and icons and a 
person with less knowledge of computers tends to be 
allergic to even touching a keyboard/mouse. 
[0007] Therefore, it is a great problem that such data so 
processors are very difficult to handle for those who 
have less knowledge of computers. 
[0008] It is therefore desirable to provide an apparatus 
which can easily manipulate an object displayed on a 
display unit. 55 
[0009] It is also desirable to provide a user interface 
with which a user can easily manipulate an object dis- 
played on a display unit. 



[0010] IBM Technical Disclosure Bulletin, vol. 33, No. 
1B, June 1990, pages 277-278 discloses an apparatus 
according to the preamble of accompanying claim 1. In 
this prior art, icons resembling three-dimensional push 
buttons are displayed on the panel (screen). These 
icons can be "manipulated" by pushing them in or out, 
the icon display changing in appearance to reflect the 
amount of depression of the push button. There is only 
one type of object (icon) and they are displayed at fixed 
positions on the screen. 

[001 1 ] According to the present invention, there is pro- 
vided an apparatus for use with a display device to 
manipulate an object displayed on the display device, 
the apparatus comprising: 

a touch panel, mounted to the display device and 
representing a display surface of the display device, 
which is sensitive to characteristics of a touching 
contact on the touch panel, the characteristics 
including at least co-ordinate positions of the touch- 
ing contact, for outputting corresponding touch 
panel information representing the characteristics 
of the touching contact on the touch panel; and 
object manipulation means for manipulating and 
displaying the object of the display device in 
response to said touch panel information; 
characterised in that said object manipulation 
means comprises: 

object storage means for storing at least one data 
file which stores object data for displaying the 
object; 

display information storage means for storing dis- 
play position information which specifies the posi- 
tion where the object is displayed on the display 
device; and 

display control means for recognizing the type of 
manipulation of the object in accordance with the 
touch panel information indicating a movement 
such that two touching contacts touch both sides of 
the object, move and stop with a distance in 
between, and for controlling to display the object on 
the display device so that the object moves on the 
display device from where the two touching con- 
tacts touch both sides of the object to where the two 
touching contacts stop with a distance in between. 

[0012] According to another aspect of the invention, 
there is provided a controller for controlling the above- 
identified apparatus and comprising: 

outputting means for outputting touch panel infor- 
mation representing a position of the touching con- 
tact; and 

object manipulation means for manipulating and 
displaying the object on the display device in 
response to said touch panel information; 
characterised in that said object manipulation 
means comprises: 
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object storage means for storing at least one data 
file which stores object data for displaying the 
object; 

display information storage means for storing dis- 
play position information which specifies the posi- 
tion where the object is displayed on the display 
device; and 

display control means for recognizing the type of 
manipulation of the object in accordance with the 
touch panel information indicating a movement 
such that two touching contacts touch both sides of 
the object, move and stop with a distance in 
between, for controlling to display the object on the 
display device so that the object moves on the dis- 
play device from where the two touching contacts 
touch both sides of the object to where the two 
touching contacts stop with a distance in between. 

[0013] Thus, the present invention can provide a 
touch-sensitive panel (e.g. touch screen), means stor- 
ing a plurality of data files, display information storage 
means and display control means, which can be used 
with a display device of a computer system or worksta- 
tion. 

[0014] Reference is made, by way of example, to the 
accompanying drawings, in which Figures 5 and 6 par- 
ticularly concern the present invention, and Figures 7 to 
1 1 relate to other types of manipulation illustrated by 
way of background explanation. 

Fig. 1 illustrates a computer system with a conven- 
tional user interface; 

Fig. 2 is a configuration diagram of a touch-screen- 
equipped workstation, to which the present inven- 
tion may be applied; 

Fig. 3 is a schematic diagram illustrating the princi- 
ple of the present invention; 
Fig. 4(a) shows a display information table; 
Fig. 4(b) shows touch screen information; 
Fig. 5 is a flowchart illustrating a pick manipulation; 
Fig. 6 is a diagram illustrating a pick manipulation; 
Fig. 7 is a diagram illustrating a scroll manipulation; 
Fig. 8 is a diagram illustrating a push manipulation; 
Fig. 9 is a diagram illustrating a flip manipulation; 
Fig. 10 is a diagram illustrating a roll manipulation; 
Fig. 11 is a diagram illustrating a distort-restore 
manipulation. 

[0015] Throughout the above-mentioned drawings, 
identical reference numerals are used to designate the 
same or similar component parts. 
[0016] Fig. 2 is a configuration diagram of a touch 
screen-equipped workstation for implementing the 
present invention. 

[001 7] In addition to the conventional system shown in 
Fig. 1 , the system includes an input-output (abbreviated 
to I/O) port 8, a touch screen controller 15 and a touch 
screen unit 1 with a touch screen 1 1 . The touch screen 



controller 15, connected to the input-output port 8 
through an RS-232C interface, controls the touch 
screen unit 1. The touch screen unit 1, which is sensi- 
tive to a position or positions (X-Y co-ordinates) where it 
5 is touched, and preferably also to a pressure applied to 
it, acts as a user interface that allows a user to send sig- 
nals to the CPU by touching an area thereon with a 
body, such as a finger or a pencil. 

[0018] Fig. 3 is a schematic diagram illustrating the 

10 principle of the present invention. 

[0019] For easy understanding of the principle, the 
input-output port 8, touch screen controller 15 and 
touch screen unit 1 shown in Fig. 2 are represented by 
the touch screen unit 1 ; and the frame memory 6 and 

is display unit 3 are represented by the display unit 3. A 
system controller 50, touch discriminator 51 display 
controller 52 and display information table 1T, which are 
stored in the main memory 5, control display operations 
featured by the present invention. 

20 [0020] Fig. 4(a) shows a display information table. Fig. 
4(b) shows touch screen information. 
[0021] A display information table 1T, which is pro- 
vided in the main memory 5, corresponding to objects, 
includes an object type, display position information, file 

25 information, normal-display file name and special-state 
file name. The object type defines the type including the 
shape, properties, circumstances, etc., of the object. 
The display position information defines the size of the 
object (width, height), and the position (top-left coordi- 

30 nates X,Y) and the angle at which the object is dis- 
played on the display unit 1 . 

[0022] The file information stored in the display data 
file, which is used for an object which is so large in size 
that it requires scrolling to view the whole object, 

35 defines the size (width W, height H) of the whole object 
relative to the display screen size, and also the position 
(top-left co-ordinates X, Y) of the object portion being 
displayed on the display device, relative to the whole 
object data. The normal display file name specifies a 

40 normal display file where object data for displaying a 
normal-state of the object is stored. The special-state 
file name specifies a special-state file where object data 
for displaying a special-state (e.g. turn-over indication of 
display color, used for displaying intermediate process 

45 of manipulating the object) of the object is stored. 

[0023] Touch-screen information 2 1, which is sent from 
the touch screen unit 1 , includes a touch position (X-Y 
coordinates) where the touch screen 1 1 is touched and 
a pressure applied thereon. The following manipula- 

so tions are possible, in which the Pick Manipulation (2) is 
particularly relevant to the present invention. 

(1) The touch discriminator 51 , based on the touch 
screen information 2 1 from the touch screen unit 1 , 
55 discriminates the type of a touch an operator's fin- 
ger has on the touch screen 1 1 , that is, a touch type 
including, i.e., a "continuous touch start" and "con- 
tinuous touch end" explained later. The touch dis- 
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criminator 51 sends to the system controller 50, the 
result of the discrimination as a touch report 3R, 
which includes a touch type and touch coordinates. 

Based on the touch report 3R from the touch 
discriminator 51 and the display information table s 
1T, the system controller 50 determines the type of 
a manipulation conducted by an operator and, 
according to the determination, updates the display 
information table 1T. Then, the system controller 50 
sends to the display controller 52, a display update 10 
request 4Q along with "display update data" which 
includes contents of the display information table 
1T updated (including display position information, 
file information, normal display file name and spe- 
cial-state file name). 15 

On receipt of the display update request 4Q 
from the system controller 50, the display controller 
52 reads display file data (including object data) 
specified by the file name from the hard disk 7 and 
stores the data into the main memory 5. The display 20 
controller 52 then updates the object data in 
accordance with the display update data from the 
system controller 50 and loads the thus-updated 
object data into the frame memory 6 to display the 
object, as manipulated by the operator on the touch 25 
screen unit 1 . 

Thus, the present invention determines a 
manipulation to be conducted on the object dis- 
played, based on the touch screen information 21 
which results from an operator's touching the touch 30 
screen 11 and the display information table 1T 
which defines the object's shape, physical proper- 
ties, display position, etc. It then displays the object 
according to the manipulation determined, as 
intended by the operator. 35 
(2) Pick manipulation (see Figs. 5 and 6.) 

A pick manipulation is conducted in such a way 
as an object is picked up at a position on the display 
surface of the display unit 3 and placed at another 
position. 40 

Fig. 5 is a flowchart illustrating a pick manipula- 
tion. Fig. 6 is a diagram illustrating a pick manipula- 
tion. 

A pick manipulation is carried out according to 
the following steps (S1-S8) in Fig. 5: 45 

(51) The system controller 50 receives a touch 
report 3R from the touch discriminator 51 . 

(52) The system controller 50 checks the touch 
report 3R to see whether the object-finger rela- 50 
tion is a pick manipulation as shown in Fig. 
6(a), based on the touch report 3R and con- 
tents of the display information table 1T shown 

in Fig. 6(c). When the relation is not a pick 
manipulation, the system controller 50 checks 55 
the touch report 3R for other manipulation. 

(53) When the relation is a pick manipulation, 
the system controller 50 sends a display 



update request 4Q including "display update 
data", commanding that the special-state file 
(turn-over indication) be displayed at the posi- 
tion specified by the display information table 
1T. 

(54) The system controller 50 receives a touch 
report 3R. 

(55) The system controller 50 determines 
whether the touch report 3R includes a "contin- 
uous touch end", which occurs when the finger- 
object relation is as in Fig. 6(b). When a "con- 
tinuous touch end" is reported, the operation 
goes to step (S8). 

(56) Otherwise, the system controller 50 
updates the display position information "coor- 
dinates (X, Y)" of the display information table 
1T so that the object is positioned between the 
two fingers. 

(57) The system controller 50 sends display 
update request 4Q to the display controller 52, 
commanding that the special-state file be dis- 
played according to the display information 
table 1T updated, and returns to step (S4). 

(58) When "continuous touch end" is reported 
by a touch report 3R, the system controller 50 
sends a display update request 4Q to the dis- 
play controller 52, commanding that the nor- 
mal-display file be displayed at the position 
specified in the display information table 1T. 

Following manipulations are carried out in the 
same way as described in the above flowchart of 
the pick manipulation. 
(3) Scroll manipulation (see Fig. 7.) 

A scroll manipulation is conducted in such a 
way as an object extending outside of the display 
surface of the display unit 3 is moved into and out of 
the display surface. 

Fig. 7 is a diagram illustrating a scroll manipu- 
lation. 

On determining that the finger moves while 
touching the touch screen 1 1 based on the touch 
screen information 21 from the touch screen unit 1, 
the discriminator 51 sends to the system controller 
50, a touch report 3R including "continuous touch 
start" for the touch type and also "coordinates (800, 
800)" for the touch position. As another touch 
screen information 21 comes in, the discriminator 51 
sends a touch report 3R including "continuous 
touch in progress" and coordinates (780, 800). 
When the touch screen information 2 1 is not sent for 
more than 100 milliseconds, for example, the dis- 
criminator 51 sends a touch report 3R including 
"continuous touch end" and coordinates (700, 800) 
to the system controller 50. 

when a "continuous touch start" is reported and 
the "object type" is defined as "out-screen" in the 
display information table 1T, the system controller 
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50 recognizes the object as a large one extending 
beyond the display screen. Then, the system con- 
troller 50 determines the speed at which the finger 
has moved from right to left, for example, based on 
a change in the X-coordinate in the touch report 3R. 5 

Depending on whether the finger has moved at 
a speed of more (high-speed) or less (normal- 
speed) than for example 20 dots (pixels) e.g. since 
the last check, the display screen is scrolled first at 
an interval of 100 or 500 milliseconds, respectively. 10 
Then, the interval, at which the display update 
request 4Q is sent to the display controller 52, is 
increased by a factor 1.5 at each touch report 3R 
and, when the interval reaches 2 seconds, the 
scrolling is stopped. 15 

Practically, the screen is so controlled that it 
starts scrolling at an above-mentioned speed after 
a finger has moved a distance of 4 dots or more. 
That is, on recognizing that the finger has moved for 
that distance, the system controller 50 updates the 20 
file information "display position X" of the display 
information table 1T so that the object is displayed 
to the left by 10 dots, for example. Then, it sends to 
the display controller 52, a display update request 
including display position information, file informa- 25 
tion and normal display file name from the display 
information table 1T updated. 

The display controller 52 reads from the hard 
disk a display file specified by the normal display file 
name and loads it in the main memory 5. The dis- 30 
play controller 52 then transfers only the part of the 
display file specified by the file information "display 
position X" of the display information table 1T, from 
the main memory 5 to the appropriate location of 
the frame memory 6. 35 

In the same way, the system controller 50 
sends a display update request 4Q to the display 
controller 52 every time it receives a touch report 
3R . 

When another "continuous touch" is reported 40 
before the scroll currently in progress comes to a 
stop, a new scroll can start from this point and at the 
first speed described above. 

(4) Scroll-stop manipulation (see Fig. 7.) 

Fig. 7 is a diagram illustrating a scroll manipu- 45 
lation. 

When a touch position given by a touch report 
3R is the same as or approximately 5 dots apart 
from the position of the scrolling currently in 
progress, the system controller 50 doubles the fre- so 
quency with which display update requests 4Q are 
sent to the display controller 52, in order to put an 
end to the scrolling. 

(5) Push manipulation (see Fig. 8.) 

A push manipulation is conducted in such a 55 
way as an object is pushed on the display surface of 
the display unit 3. Fig. 8 is a diagram illustrating a 
push manipulation. 



The system controller 50 determines the type 
of a manipulation, based on the touch report 3R 
and contents of the display information table 1T 
shown in Fig. 8(c). When the manipulation is a push 
manipulation as shown in Fig. 8(a), the system con- 
troller 50 sends to the display controller 52, a dis- 
play update request 4Q including display position 
information, file information and normal display file 
name so that the object is displayed close to the fin- 
ger position reported by the touch report 3R. The 
above display operation is repeated until a "contin- 
uous touch end" is reported by a touch report 3R. 

(6) Push-while-rotate manipulation (see Fig. 8.) 

A push-while-rotate manipulation is conducted 
in such a way as an object is pushed at a position 
off its center (or the center of gravity) and it moves 
rotating on the display surface of the display unit 3. 

Fig. 8 is a diagram illustrating a push manipula- 
tion. 

The system controller 50 determines the type 
of a manipulation, based on the touch report 3R 
and contents of the display information table 1T 
shown in Fig. 8(c). When the manipulation is a 
push-while-rotate manipulation as shown in Fig. 
8(b), the system controller 50 sends to the display 
controller 52, display update requests 4Q with the 
angle of rotation increasing by 2 degrees, i.e., while 
increasing the angle in the display information table 
1T shown in Fig. 8(c). 

The display controller 52 reads the display file 
from the hard disk and loads the data in the main 
memory 5, rotates the object by the angle and with 
the left-top coordinates (X, Y) as a rotational center, 
as specified by the display update request 4Q, and 
transfers the data with the object rotated, from the 
main memory 5 to the frame memory 6. 

(7) Flip manipulation (see Fig. 9.) 

A flip manipulation is conducted in such a way 
as a finger flips an object or touches the object from 
a remote position at a high speed on the display 
surface of the display unit 3. 

Fig. 9 is a diagram illustrating a flip manipula- 
tion. 

When a touch report 3R is input from the touch 
discriminator 51 , the system controller 50 descrimi- 
nates the type of a manipulation based on the touch 
report 3R and contents of the display information 
table 1T shown in Fig. 9 (c). When the manipulation 
is a flip manipulation as shown in Fig. 9 (a), the sys- 
tem controller 50 obtains a finger speed based on 
the touch report 3R and also an object speed (i.e., 
the interval at which display update requests 4Q 
are sent to the display controller 52), in the same 
way as described in item (3). The system controller 
50 sends display update requests 4Q to the display 
controller 52, while updating the display position 
information left-top coordinates (X, Y) of the display 
information table 1T so that the object moves in the 
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direction the finger moves. The system controller 50 
stops moving the object when the above-mentioned 
interval reaches 2 seconds. 

(8) Flip-under-gravity manipulation (see Fig. 9.) 

A flip-under-gravity manipulation is conducted s 
in such a way as an object which is subjected to a 
gravity is flipped by a finger on the display surface 
of the display unit 3. 

Fig. 9 is a diagram illustrating a flip manipula- 
tion. 10 

When the finger manipulation is a flip as in the 
above item (8) and the display information table 1T 
defines the object type as "gravity" meaning that the 
object is subjected to gravity, for example, the 
object moves under the combined influences of is 
inertia and simulated gravity, i.e. "falls" as shown in 
Fig. 9(b). Therefore, the system controller 50 sends 
display update requests 4Q to the display controller 
52, while updating the display position information 
left-top coordinates (X, Y) by adding a value to the 20 
Y-coordinate of of the display information table 1T. 
The value is represented by 2 to the Nth power (N: 
the number of display update requests 4Q sent). In 
this case, too, the system controller 50 stops mov- 
ing the object when the above-mentioned interval 25 
reaches 2 seconds. The resulting trajectory may be 
a parabola. 

(9) Roll manipulation (see Fig. 10.) 

A roll manipulation is conducted in such a way 
as a rollable object is rolled by a finger on the dis- 30 
play surface of the display unit 3. 

Fig. 10 is a diagram illustrating a roll manipula- 
tion. 

When a touch report 3R is input from the touch 
discriminator 51 and the display information table 35 
1T defines the object type as "rollable" meaning 
that the object is constructed such that it rolls when 
flipped like a globe or a cylinder, as shown in Fig. 
10(a), the system controller 50 sends display 
update requests 4Q to the display controller 52, 40 
while updating the display position information left- 
top coordinates (X, Y) of the display information 
table 1T so that the object moves a distance 10 per 
cent behind the distance and in the direction the fin- 
ger moves. 45 

(10) Distort-restore manipulation (see Fig. 11.) 

A distort-restore manipulation is conducted in 
such a way as an"elastic"object is pressed by a fin- 
ger on the display surface of the display unit 3, 
thereby deforming the displayed object. so 

Fig. 1 1 is a diagram illustrating a distort-restore 
manipulation. 

When a touch report 3R is input from the touch 
discriminator 51 and the display information table 
1 T defines the object type as "elastic" meaning that 55 
the object can be distorted and restored according 
to a pressure applied thereon by a finger, as shown 
in Fig. 1 1(a), the system controller 50 calculates an 



amount of distortion of the object based on the 
pressure reported by the touch report 3R. It stores 
in the display information table 1T, a special-state 
file name specifying one of special-state files (for 
displaying a distorted state of the object in turn-over 
indication) corresponding to the amount of distor- 
tion calculated. Then, the system controller 50 
sends a display update request 4Q to the display 
controller 52, commanding that the special-state file 
be displayed at the current display position. When 
the above operation is repeated as necessary and 
a "continuous touch end" is reported by a touch 
report 3R, the system controller 50 sends a display 
update request 4Q (with a normal display file name 
specified) to the display controller 52, commanding 
that a normal display file (normal indication) be dis- 
played at the current display position. A plurality of 
special-state files are provided in the hard disk 7, 
corresponding to the amount of distortion of the 
object, which results from a pressure applied on the 
touch screen 1 1 . 

[0024] As is apparent by the above description, the 
present invention regards a display screen as a virtual 
space. It defines conditions and physical properties of 
an object (e.g., weight, hardness, frictional resistance, 
center of gravity) in the display information table 1T. It 
also receives touch screen information 21 indicating a 
finger-touched position and pressure is input from a 
touch screen unit 1 . Based on the touch screen informa- 
tion 21 and the display information table 1T, the present 
invention determines a manipulation to be conducted on 
the object displayed, e.g. scrolling, picking (up), push- 
ing, rolling, distorting the object on the display surface of 
the display unit 3. Thus, the present invention allows a 
user to manipulate an object displayed on a display 
device quite easily, even when the user has little knowl- 
edge of computers. 

Claims 

1. An apparatus for use with a display device to 
manipulate an object displayed on the display 
device, the apparatus comprising: 

a touch panel, mounted to the display device 
and representing a display surface of the dis- 
play device, which is sensitive to characteris- 
tics of a touching contact on the touch panel, 
the characteristics including at least co-ordi- 
nate positions of the touching contact, for out- 
putting corresponding touch panel information 
representing the characteristics of the touching 
contact on the touch panel; and 
object manipulation means for manipulating 
and displaying the object of the display device 
in response to said touch panel information; 
characterised in that said object manipulation 
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means comprises: 

object storage means for storing at least one 
data file which stores object data for displaying 
the object; 

display information storage means for storing s 
display position information which specifies the 
position where the object is displayed on the 
display device; and 

display control means for recognizing the type 
of manipulation of the object in accordance 10 
with the touch panel information indicating a 
movement such that two touching contacts 
touch both sides of the object, move and stop 
with a distance in between, and for controlling 
to display the object on the display device so is 
that the object moves on the display device 
from where the two touching contacts touch 
both sides of the object to where the two touch- 
ing contacts stop with a distance in between. 

20 

2. A controller for controlling an apparatus according 
to claim 1 , the controller comprising: 

outputting means for outputting touch panel 
information representing a position of the 25 
touching contact; and 

object manipulation means for manipulating 
and displaying the object on the display device 
in response to said touch panel information; 
characterised in that said object manipulation 30 
means comprises: 

object storage means for storing at least one 
data file which stores object data for displaying 
the object; 

display information storage means for storing 35 
display position information which specifies the 
position where the object is displayed on the 
display device; and 

display control means for recognizing the type 
of manipulation of the object in accordance 40 
with the touch panel information indicating a 
movement such that two touching contacts 
touch both sides of the object, move and stop 
with a distance in between, for controlling to 
display the object on the display device so that 45 
the object moves on the display device from 
where the two touching contacts touch both 
sides of the object to where the two touching 
contacts stop with a distance in between. 

50 

3. The apparatus of claim 1 or 2, wherein 

said display control means is operable to rec- 
ognize the movement as a pick manipulation 
and to perform control so as to display the 55 
object on the display device so that the object 
moves on the display device from where the 
two touching contacts touch both sides of the 



object to where the two touching contacts stop 
with a distance in between. 
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